feat: Store only the last row in the previous cursor for round robin tie-breaking purposes#23619
Open
ariel-miculas wants to merge 5 commits into
Open
feat: Store only the last row in the previous cursor for round robin tie-breaking purposes#23619ariel-miculas wants to merge 5 commits into
ariel-miculas wants to merge 5 commits into
Conversation
EmilyMatt
reviewed
Jul 16, 2026
| /// any buffer the cursor's [`CursorValues`] holds. Used to remember a | ||
| /// partition's last row across a batch boundary without keeping the | ||
| /// whole exhausted batch's memory alive (see [`Self::is_eq_to_prev_one`]). | ||
| pub fn last_value(&self) -> T::SingleRowValue { |
Contributor
There was a problem hiding this comment.
I feel like this should return an Option, in case values() is empty
Contributor
Author
There was a problem hiding this comment.
values() shouldn't be empty, there are some asserts in the code:
assert!(rows.num_rows() > 0);
assert!(array.len() > 0, "Empty array passed to FieldCursor");
I don't imagine a Cursor with empty values being useful.
impl RowValues {
/// Create a new [`RowValues`] from `rows` and a `reservation`
/// that tracks its memory. There must be at least one row
///
/// Panics if the reservation is not for exactly `rows.size()`
/// bytes or if `rows` is empty.
pub fn new(rows: Arc<Rows>, reservation: MemoryReservation) -> Self {
assert_eq!(
rows.size(),
reservation.size(),
"memory reservation mismatch"
);
assert!(rows.num_rows() > 0);
Self {
rows,
_reservation: reservation,
}
}
}
pub fn new<A: CursorArray<Values = T>>(
options: SortOptions,
array: &A,
reservation: MemoryReservation,
) -> Self {
assert!(array.len() > 0, "Empty array passed to FieldCursor");
let null_threshold = match options.nulls_first {
true => array.null_count(),
false => array.len() - array.null_count(),
};
Contributor
Author
There was a problem hiding this comment.
The previous code also assumed values is never empty:
fn is_eq_to_prev_row_in_prev_batch(&self, other: &Self) -> bool {
assert_eq!(self.offset, 0);
T::eq(
&self.values,
self.offset,
&other.values,
other.values.len() - 1,
)
}
Contributor
Author
|
@Dandandan wondering whether we can change ReusableRows to only hold one slot after this change, since the previous cursor will no longer need to be kept alive for the round robin tie-breaking feature. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
See the linked issue.
Note that this PR also contains the changes in #23606, so this will have to be rebased
What changes are included in this PR?
Store only the last row in prev_cursors instead of keeping the entire cursor
Are these changes tested?
Added a test to show the improvement
Are there any user-facing changes?
No